High-density, fail-in-place switches for computer and data networks

ABSTRACT

A structure for a network switch. The network switch may include a plurality of spine chips arranged on a plurality of spine cards, where one or more spine chips are located on each spine card; and a plurality of leaf chips arranged on a plurality of leaf cards, wherein one or more leaf chips are located on each leaf card, where each spine card is connected to every leaf chip and the plurality of spine chips are surrounded on at least two sides by leaf cards.

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

This invention was made with United States Government support undercontract B601996 awarded by U.S. Department of Energy. The Governmenthas certain rights to this invention.

BACKGROUND

The present invention generally relates to network switches, and moreparticularly to the architecture and structure of a network switchhaving leaf cards surrounding spine cards on at least two sides.

Modern computer systems have sophisticated networks, either custom orbased on evolving network standards. Although some systems haveprocessing nodes that are self-interconnected (e.g., the IBM Blue Geneline of computers), most systems are based on cabling the computing orstorage elements of the system into a network comprised of externalswitches or routers. In the industry, a switch connects a set ofnetworked components into a local-area network, whereas a routerconnects sets of local-area networks. The places where cables areattached are called ports. For the present invention, switches androuters may be referred to as network switches.

Switches are based on switch chips. Switch chips for Ethernet orInfiniband, two popular standards, are typically built of CMOS. Forexample, an Ethernet switch comprising 48 places, with each placeallowing data to be sent and received at a data rate of10-Gigabits-per-second, is referred to as a “48-port, 10-Gb/s switch”.Another example is a 36-port, 56-Gigabit-per-second-per-port Infinibandswitch, which has 36 places where data can be sent and received as 4parallel lanes, each at 14 Gb/s.

SUMMARY

According to one embodiment of the present invention, a structure for anetwork switch is provided. The structure may include a plurality ofspine chips arranged on a plurality of spine cards, wherein one or morespine chips are located on each spine card; and a plurality of leafchips arranged on a plurality of leaf cards, wherein one or more leafchips are located on each leaf card, wherein each spine card isconnected to every leaf chip and the plurality of spine chips aresurrounded on at least two sides by leaf cards.

According to another embodiment of the present invention, a structurefor a network switch is provided. The structure may include a set ofleaf assemblies connected to a set of spine assemblies, the leafassemblies include a leaf card and leaf chips, the spine assembliesinclude a spine card and spine chips, wherein every leaf assembly isconnected to every spine assembly and the leaf assemblies surround thespine assemblies on at least two sides.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

The following detailed description, given by way of example and notintended to limit the invention solely thereto, will best be appreciatedin conjunction with the accompanying drawings, in which:

FIG. 1 is an illustration of prior art showing the orientation of a twolayer network.

FIGS. 2A and 2B are illustrations of prior art showing a modern coreswitch.

FIG. 3 is an illustration of prior art showing the orientation of athree layer network.

FIGS. 4A-4E are illustrations of a spine-and-leaf array, according to anembodiment.

FIGS. 5A-5E are illustrations of a single spine assembly, according toan embodiment.

FIGS. 6A-6F are illustrations of leaf assemblies, according to anembodiment.

FIGS. 7A-7F are illustrations of spine-and-leaf array and a mechanicaland cooling infrastructure, according to an embodiment.

FIGS. 8A-8C are illustrations of a standoff structure, according to anembodiment.

FIGS. 9A-9E are illustrations of methods of attaching two or more cards,according to an embodiment.

FIGS. 10A and 10B are illustrations of a liquid-cooled spine array,according to an alternative embodiment.

FIGS. 11A-14B are illustrations of components of the spine-and-leafarray, according to an alternative embodiment.

The drawings are not necessarily to scale. The drawings are merelyschematic representations, not intended to portray specific parametersof the invention. The drawings are intended to depict only typicalembodiments of the invention. In the drawings, like numbering representslike elements.

DETAILED DESCRIPTION

Detailed embodiments of the claimed structures and methods are disclosedherein; however, it can be understood that the disclosed embodiments aremerely illustrative of the claimed structures and methods that may beembodied in various forms. This invention may, however, be embodied inmany different forms and should not be construed as limited to theexemplary embodiments set forth herein. Rather, these exemplaryembodiments are provided so that this disclosure will be thorough andcomplete and will fully convey the scope of this invention to thoseskilled in the art. In the description, details of well-known featuresand techniques may be omitted to avoid unnecessarily obscuring thepresented embodiments.

References in the specification to “one embodiment”, “an embodiment”,“an example embodiment”, etc., indicate that the embodiment describedmay include a particular feature, structure, or characteristic, butevery embodiment may not necessarily include the particular feature,structure, or characteristic. Moreover, such phrases are not necessarilyreferring to the same embodiment. Further, when a particular feature,structure, or characteristic is described in connection with anembodiment, it is submitted that it is within the knowledge of oneskilled in the art to affect such feature, structure, or characteristicin connection with other embodiments whether or not explicitlydescribed.

For purposes of the description hereinafter, the terms “upper”, “lower”,“right”, “left”, “vertical”, “horizontal”, “top”, “bottom”, andderivatives thereof shall relate to the disclosed structures andmethods, as oriented in the drawing figures. The terms “overlying”,“atop”, “on top”, “positioned on” or “positioned atop” mean that a firstelement, such as a first structure, is present on a second element, suchas a second structure, wherein intervening elements, such as aninterface structure may be present between the first element and thesecond element. The term “direct contact” means that a first element,such as a first structure, and a second element, such as a secondstructure, are connected without any intermediary conducting, insulatingor semiconductor layers at the interface of the two elements.

In the interest of not obscuring the presentation of embodiments of thepresent invention, in the following detailed description, someprocessing steps or operations that are known in the art may have beencombined together for presentation and for illustration purposes and insome instances may have not been described in detail. In otherinstances, some processing steps or operations that are known in the artmay not be described at all. It should be understood that the followingdescription is rather focused on the distinctive features or elements ofvarious embodiments of the present invention.

Referring now to FIG. 1 (prior art), a focus is on very large switches(i.e., core switches or director switches). The number of ports in atypical core switch can vastly exceed the number of ports in a switchchip. A popular configuration is a two-layer network, where thetwo-layer network is arranged in a spine-leaf topology, as illustrated.

Each of a first set of switch chips is called a leaf chip and each leafchip has p_(L) ports. Four leaf chips are illustrated and each leaf chiphas four ports (i.e., p_(L)=4). Typically, in a core switch, the leafchips are placed on a plurality of circuit cards, often called linecards or leaf cards. Generally, each leaf chip has externally facingports equal to p_(L)/2 connected on a first edge of the leaf card tocables running to external computer elements (i.e., nodes), such as, forexample, a processor or a storage node. There also may be internallyfacing ports equal to p_(L)/2 connected through leaf-to-spine connectorslocated on a second edge of the leaf card to a second set of switchchips called spine chips, such as, for example, spine chip S1 and spinechip S2. The spine chips are illustrated as having p_(S) ports.Typically, spine chips and leaf chips are identical, wherep_(S)=p_(L)=p. One or more spine chips are typically placed on a circuitcard called a spine card. Every port of every spine chip is connected toan internally facing port of a leaf chip. Consequently, the maximumnumber of externally facing ports (P_(ext)) in a two-layer core switchmay be computed as follows.

Let:N _(L)≡Number of leaf chips in the core switch  (1)N _(S)≡Number of spine chips in the core switch  (2)p _(L)≡Number of ports per leaf chip  (3)p _(S)≡Number of ports per spine chip  (4)P _(ext)≡Total number of externally facing ports for the coreswitch  (5)

Because one port of each spine chip must connect to a port on a leafchip, it follows that:N _(L) =p _(S)  (8)

For a non-blocking switch, only half of the leaf chip ports areexternally facing, it follows that:P _(ext) =N _(L) *p _(L)/2  (7)

Substituting (6) into (7) yields:P _(ext) =p _(S) *p _(L)/2  (8)

When spine chips are identical to leaf chips, it follows that:p _(S) =p _(L) ≡p  (9)P _(ext) =p ²/2  (10)

Because the total number of internally facing ports on the leaf chips,N_(L)*p_(L)/2, must equal the total number of ports N_(S)*p_(S) on thespine chips, it follows also that:N _(S) =N _(L) *p _(L)/(2*p _(S))  (11)

whence, for the special case (9), it follows that:N _(S) =N _(L)/2  (12)

For the example above, where the special case (9) with p=48, anon-blocking core switch has P_(ext)=1152 externally facing portsaccording to equation (10).

Referring now to FIGS. 2A and 2B, a modern core switch is housed in anair-cooled rack, where the modern core switch has 1152 externally facingports. Leaf cards may be connected on a first side of the rack (e.g., afront side) and spine cards may be connected on a second side of therack (e.g., a back side). The spine cards may be vertical while the leafcards may be horizontal, but other orientations may be used. This may bea typical approach because every leaf chip has a connection to everyspine chip.

A limitation of the current approach is that the maximum value ofP_(ext), that can be economically packaged, is often smaller than thenumber of devices desired for interconnection. For example, economicpackaging of a 10GBASE-T Ethernet is accomplished using inexpensive RJ45connectors and low-cost category-6 electrical cables, but theseinexpensive components are bulky, so a small number of leaf cards canaccommodate only a modest value of P_(ext). To achieve larger values ofP_(ext), expensive alternative components are often used, such as aQSFP. A QSFP can accept four-channel-wide optical cables, which, for 10GEthernet, must be connected to computing or storage nodes usingexpensive 1-to-4 optical breakout cables.

The fact that P_(ext) is limited by packaging constraints is furtheraggravated by the development of new, faster switch chips with larger pvalues. The higher speed of the port can be handled by appropriate IOcell design, proper high-speed electrical packaging techniques, and useof commensurately faster cables. However, because the number ofexternally facing ports P_(ext) in a two-level fat-tree is quadratic inp, as stated by equation (10), the problem of packaging a core switchbecomes dramatically more difficult as p increases.

In some switches, a port is implemented by a single electrical channel.For example a 10 Gb/s Ethernet port might be made by two differentialwiring pairs, one receiving data and one sending data, carried onso-called twin-axial electrical cable. In other switches, a port may beimplemented optically: using optical transceivers, the same 10 Gb/ssignals may be sent over a pair of optical fibers, one sending data andone receiving data. Such a pair of signals is called a lane. A port canalso include multiple lanes. If the signaling rate is 25 Gb/s per lane,these four lanes can deliver, in parallel, 100 Gb/s per port. There is anew Ethernet standard that also uses 4 parallel lanes at 25 Gb/s.Assessment of the limits of packaging and chip technology suggests that64 port chips, each with four lanes, are possible with today'stechnologies.

Referring now to FIG. 3, packaging constraints as P_(ext) increasesbecome even more acute for a three-level network, where p=8. For suchthree-level networks, the number of externally facing ports can be tensof thousands. For example, with p=48, a switch that implements such athree-level network (m=3) has 2(p/2)^(m)=27,648 externally facing ports.

Equation (10) applies for non-blocking switches, in which only half of aleaf switch's ports are externally facing, such that all cable ports maysend or receive data at full bandwidth without slowdown. A more popularstructure includes oversubscribed switches, in which there are more leafcards than there are spine cards to support them at full bandwidth. Inother words, the fraction of a leaf switch's ports that are externallyfacing is greater than ½.

Let:f=Fraction of a leaf switch's ports that are externally facing.

Then equation (7) generalizes to:P _(ext) =f*N _(L) *p _(L)  (13)

Substituting (6) into (13) yields:P=f*p _(S) *p _(L)  (14)

Consequently, for the typical case (9) (i.e., p_(S)=p_(L)≡p), it followsthat:P _(ext) =f*p ²  (15)

Because the total number of internally facing ports on the leaf chips,(1−f)*N_(L)*p_(L), must equal the total number of ports N_(S)*p_(S) onthe spine chips, it follows also that:S=(1−f)*N _(L) *p _(L) /p _(S)  (17)

whence, for the special case (9), it follows that:N _(S)=(1−f)*N _(L)  (17)

In an example, f=¾. Then, for special case (9), P_(ext)=0.75*p². fgreater than ½ may yield data collisions if more than ½ of the nodesconnected to the leaf cable ports want to send or receive data at once.However, such collisions seldom, if ever, occur, accounting for thepopularity of the over-subscribed switches.

As indicated by equation (15), the number of externally facing portsP_(ext) for over-subscribed switches (f>½) is greater than stated by theprevious examples of non-blocking switches (f=½). For example, for p=48with f=¾, equation (15) gives P_(ext)=1728.

Today there are data centers with tens of thousands of nodes, many morenodes than can be connected by existing core switches. That situation ishandled by cascading core switches in various ways to make largerswitches. This comes at the cost of more cables and switches, more powerfor serialization-deserialization (SERDES) at each cable connection, andadditional latency to move through the extra switches, cables, andSERDES stages.

In some switches the spine and leaf cards can be concurrentlymaintained, and there can be an extra spine card. If a spine cardsfails, data is routed to other spine cards, and the faulty spine cardcan be replaced without turning off the switch, which continues to routedata. If a leaf card fails then the compute or storage nodes connectedto them are lost from the network, but modern data centers are designedto tolerate the failure of a compute or storage node, so that istypically acceptable.

A new packaging architecture for obtaining very-large-port-count networkswitches and obtaining a large value of P_(ext) is described below. Thearchitecture is based on the idea that leaf-to-spine connections in afat-tree are inherently redundant. For example, in FIG. 1, which is atwo-level fat tree with p=4, N_(L)=4, N_(S)=2, P_(ext)=8, if spine chipS1 fails, all externally facing ports P1-P8 can still communicate witheach other through spine chip S2. If the number N_(S) of spine chips islarge, as in modern core switches, the throughput of the switch will beonly slightly degraded each time a spine chip fails; consequently,failure of a few spine chips is hardly noticeable, because there arestill many paths through the network.

The proposed, large-P_(ext) packaging architecture deliberately exploitsthe aforesaid redundancy: it allows spine ports or even entire spinechips to fail without replacement, a strategy called “fail in place”.Where, under normal conditions, if a port on a spine chip, or even theentire spine chip, fails, it is not repaired. Rather, traffic isredirected to the other, working spine ports. Despite the failure, thereis still connectivity between all cable ports; the throughput of theswitch is merely reduced. If N_(S) is large, this throughput reductionis minimal. For example, for the previously mentioned switch chip havingp=104 ports, there are N_(S)=p/2=52 spine chips and N_(S)*p=52*104=5408spine ports. The loss of a few of these ports, the likely cost of thelarge-P_(ext) packaging architecture proposed herein, is acceptable inexchange for the much-larger benefits afforded thereby. That is,compared to the large-P_(ext) alternative of combining smaller coreswitches to make larger ones, the large-P_(ext) architecture may providethree benefits: first, reduced power because there are fewer SERDES(serializer/deserializer) channels; second, lower latency because thereare fewer “hops through switch chips”; and third, reduced cost becausethere are fewer cables.

The large-P_(ext) packaging architecture can achieve up to a four-foldincrease in P_(ext) compared to current switches. That is, because thespine cards are never removed for the lifetime of the switch, they maybe surrounded by leaf cards on multiple sides rather than on only oneside. Thus, compared to current architecture where leaf cards areconnected to spine cards on the front only, the number of leaf chipsN_(L), and hence the number of externally facing ports P_(ext), can bedoubled if the leaf cards are connected to two edges of each spine card.Similarly, compared to current architecture, the number of externallyfacing ports P_(ext) can be quadrupled if leaf cards are connected toall four edges of each spine card. Embodiments achieving large P_(ext)by exploiting the fail-in-place strategy.

Referring now to FIG. 4A, an embodiment of a spine-and-leaf array 400with fail-in-place spines includes a central stack of n_(S) spineassemblies 402 and a peripheral array of n_(L) leaf assemblies 404,where n_(S) and n_(L) are integers. The spine assemblies 402 may includea spine card 406, such as, for example, a printed circuit board (PCB),and the leaf assemblies 404 may include a leaf card 408.

Integers n_(S) and n_(L) are related to the previously defined integersN_(S) and N_(L), where:N _(S) =n _(S) *k _(S) and N _(L) =n _(L) *k _(L)  (18)k _(S)=Number of switch chips per spine card  (19)k _(L)=Number of switch chips per leaf card  (20)

In embodiments illustrated in FIGS. 4A-9E:n _(S)=13;k _(S)=4;N _(S)=52  (21)n _(L)=104;k _(L)=1;N _(L)=104  (22)

The switch chip on both spine and leaf cards has p=104 ports. Thespine-and-leaf array 400 achieves:P _(ext) =p ²/2=5408  (23)

In the illustrated embodiment, there are 112 cards shown, 104 of whichmay be leaf assemblies 404 and the remaining cards may be power cards,spine-control cards and/or leaf-control cards. Power, spine-control, andleaf-control are redundant, such that global failure never occurs fromfailure of one component, or even from failure of several components. Inthe illustrated embodiment, if fewer than eight peripheral power andcontrol cards are needed, then peripheral cards nearest the corners ofthe array may be eliminated.

The spine-and-leaf array 400 may be the heart of a core switch, but theswitch may also include mechanical and cooling infrastructure (e.g.,water cooling infrastructure). Spine assemblies 402 are linearly arrayedat a spine-to-spine pitch H along a z-direction of coordinate system410, and the leaf assemblies 404 are circumferentially arrayed aroundthe stack of spine assemblies 402, such that each leaf assembly 404 maybe electrically or optically connected to each spine assembly 402.

In an embodiment, the central stack may include one or more centralleaf-control assemblies (not illustrated) in addition to the n_(S) spineassemblies 402. Each central leaf-control assembly may be similar insize, shape and orientation to the spine assembly 402, but may includeelectronics (instead of spine chips) for initializing and controllingthe peripheral array of leaf assemblies 404. An alternative embodiment,an initialization and control function for leaf cards may be provided byone or more peripheral leaf-control assemblies (not illustrated). Eachperipheral leaf-control assembly may be similar in size, shape andorientation to a leaf assembly 404, but may include electronics (insteadof leaf chips) for initializing and controlling the leaf assemblies 404.In the illustrated embodiments shown in FIGS. 4-9, such peripheralleaf-control assemblies may occupy some number of the eight extraperipheral-card positions discussed above following equation (23). In anembodiment, redundancy of the leaf-control function may be provided,such that global failure does not occur from failure of one component orseveral components.

Referring now to FIG. 4B, each leaf assembly 404 may be disconnectedfrom the spine assemblies 402. Two leaf assemblies 404 are illustratedas being removed to illustrate the disconnection. However, removal ofthe spine assembly 402, for example, to replace a failed spine chip,would be very laborious because all n_(L) leaf cards 408 may need to bedisconnected followed by the spine assemblies 402 to reach a targetspine card. This is a benefit to the “fail in place” concept explainedabove (e.g., no spine cards will ever be removed).

Referring now to FIGS. 4C and 4D, a section of leaf assemblies 404 arehidden to illustrate the electrical connection between the spineassemblies 402 and the leaf assemblies 404. At each of the n_(S)*n_(L)intersections between the spine card 406 and the leaf card 408,electrical connection is provided by an orthogonal header 412 affixed tothe spine card 406 and a mating orthogonal receptacle 414 affixed to theleaf card 408. When a leaf card 408 is connected, n_(S) of theseheader-to-receptacle connections are simultaneously made.

Referring now to FIG. 4E, which is a top view of the spine-and-leafarray 400, the assembly shown achieves a large number of leaf cards(n_(L)=104), as well as eight additional peripheral cards for power andmanagement, with a relatively small spine card (23″×27″=584×686 mm).Large n_(L) may be desirable to achieve large P_(ext), which may bedesirable for reasons explained above. In this embodiment, a small spinecard may be desirable for two reasons. First, the degradation ofhigh-speed electrical signals in a PCB is a strong function of thedistance that the signals must travel therein. Second, economicalmanufacture of PCBs is limited to a certain maximum size, which iscurrently about 23″ by 27″ (584×686 mm).

To achieve large n_(L) with a spine card having limited perimeter,adjacent headers 412 should be close together. However, if the headers412 were close together and leaf cards were parallel, the space betweenleaf cards would be too small to accommodate the appropriate number ofexternal connectors (e.g., dual-stacked RJ-45 connectors 416 illustratedin FIG. 4D) that must be packaged at the distal edges 418 of the leafcards. In this embodiment, the leaf cards are “flared” rather thanparallel. The leaf cards are further apart at their distal edges than attheir proximal edges 420, because there is an angle Δθ between adjacentcards. Means to achieve Δθ without compromising the perimeter of thespine card and thereby limiting n_(L). An integrated cold-plate 510 mayalso be used, described below.

Referring now to FIGS. 5A and 5B, a spine-card assembly 404 includes, inaddition to spine card 408 (illustrated in FIG. 4), an n_(L)-fold arrayof the orthogonal headers 412 to convey electrical signals between thespine card and leaf cards, a k_(S)-fold array of switch chips 504,cooling means 506 to remove heat from switch chips 504, and standoffs508 to guarantee that the spine-to-spine pitch H is achieved despitemanufacturing variations in thickness of the spine cards 408. In theillustrated embodiment, the cooling means includes the liquid-cooledcold plate 510, a supply hose 512 and a return hose 514 whose proximalends are connected thereto, and appropriate plumbing 516 to connect thedistal ends of supply hose 512 and return hose 514 to a liquid-coolingsupply and a liquid-cooling return, respectively. The cooling means 506is hidden in FIG. 5B to reveal the switch chips 504. The spine-cardassembly may further include additional elements, such as, for example,a processor and memory node (not illustrated).

Referring now to FIG. 5C, which is a top view of a spine-card assembly,an arrangement of orthogonal headers 412 is shown. As described above,this arrangement produces a leaf-card spreading angle Δθ subtendedbetween adjacent leaf cards and may not sacrifice spine-card perimeter.

Referring now to FIGS. 5D and 5E, the arrangement of orthogonal headers412 is achieved by a saw-toothed serration 518 along the perimeter ofthe spine card 406. The angle between adjacent facets of the serrationis the leaf-card spreading angle Δθ. Each orthogonal header 412 abutsone of the facets, as illustrated in FIG. 5E, which is a view from thebottom surface of the spine card 406. The saw-toothed serration 518 issuperior to simple polygonal facets, because polygonal faceting of thenorth and south edges of the card would shorten the perimeters of theeast and west edges, and vice versa. In contrast, the saw-toothed facets518 do not shorten the perimeter; consequently, they achieve the largepossible value of n_(L) for the given size of spine card.

Referring to FIGS. 6A-6D, leaf-card assembly 404 includes, in additionto the leaf card 408, an n_(S)-fold array of orthogonal receptacles 414,a ½-p_(L)-fold array of external-port connectors 416, such as the set ofdual-stacked RJ45 jacks, a k_(L)-fold array of leaf chips 602, an arrayof Phy chips 604 that connects the switch chip 602 to the physicalmedium (e.g., ethernet cables) attached to the external ports 416, atailstock 606 and EMI fingerstock 608 that attenuate the passage ofelectromagnetic energy there through, tailstock mounting brackets 610,and cooling means 612 that includes, for example, the vapor chamber 614and finned heatsink 616, which are both visible in FIG. 6D, but are bothhidden on FIG. 6B to reveal the leaf chip 602. The particular coolingarrangement shown arises from the desire to place the leaf chip 602 asclose as possible to the orthogonal receptacles 414 in order to minimizeelectrical path length in the leaf card 408. To cool the leaf chip inthis position, the vapor chamber 614 is used to move the heat with lowresistance from left to right, illustrated in FIG. 6D, thereby allowinggood conduction to a broad area of the finned heat sink 616,particularly to the right end of the finned heat sink, where its finsare taller due to the leaf-card spreading angle Δθ.

Referring now to FIG. 6E, the leaf-card assembly 404 may optionallycomprise additional elements, such as the DIMM-card assembly 618.

Referring now to FIG. 6F, a portion of the ½p_(L)-fold array ofdual-stacked RJ45 connectors 416 are shown at the distal end of severalleaf cards. Referring to FIGS. 4E, 6D, and 6F, the leaf-card spreadingangle Δθ, as well as the proximal-to-distal dimension W of the leafcard, is chosen to produce, at the distal edge 418, aleaf-card-to-leaf-card pitch that accommodates the dual-stacked RJ45connectors 416 as well as EMI-containment hardware. The latter includestailstock 606 and EMI-suppressing fingerstock 608.

Referring now to FIGS. 7A-7F, a core switch 700 includes thespine-and-leaf array 400 as well as mechanical and coolinginfrastructure. As shown in FIG. 7A-7D, this infrastructure, locatedabove and below the spine-and-leaf array 400, comprises a lowercard-guide plate 702; an upper card-guide plate 704; a plurality ofcolumns 706 each of which is affixed at a first end 708 to the lowercard-guide plate 702 and is affixed at a second end 710 to the uppercard-guide plate 704, thereby to hold the card-guide plates 702 and 704a fixed distance apart; a supply-manifold assembly 712 for distributingliquid coolant to the liquid-cooled cold plates 510 of spine-cardassemblies 402; a return-manifold assembly 716 for re-aggregating liquidcoolant after it has flowed through cold plates 510; a leaf-air inletplenum 718 into which flows a first airstream 720 whose purpose is tocool the leaf-card assemblies 404; a leaf-air outlet plenum 722 whoseinterior is maintained at a lower-than-ambient pressure by a centrifugalfan (i.e. “blower”) 724 located in an exhaust chamber 726 that exhaustsairstream 720; a liquid-to-air heat exchanger 728, illustrated in FIGS.7B-7E, wherein heat is exchanged from liquid coolant to a secondairstream 730; a plurality of axial fans 734; a pump 736; a shroud 738to convey the second airstream 730; a first plumbing assembly 740 thatconveys liquid coolant from an outlet of heat exchanger 728 to supplymanifold 712; a second plumbing assembly 742 that conveys liquid coolantfrom return-manifold assembly 716 to pump 736; and a third plumbingassembly 744 that conveys liquid coolant from pump 736 to an inlet ofheat exchanger 728. Leaf-air inlet plenum 718 is illustrated as opaquein FIG. 7A and transparent in FIGS. 7B-7E. Leaf-air outlet plenum 722 isillustrated as opaque in FIGS. 7A-7C and 7E and transparent in FIG. 7D.Exhaust chamber 726 is illustrated as opaque in FIGS. 7A-7B andtransparent in FIGS. 7C-7E. Shroud 738 is illustrated as opaque in FIG.7A and transparent in FIGS. 7B-7E.

Referring now to FIG. 7E, in which some of the leaf-card assemblies arehidden, airstream 720 flows through perforations 746 in the lowercard-guide plate 702. The vertical dimension of the leaf-air inletplenum 718 may be large enough to insure that the flow through allperforations 746 is substantially equal; that is, enclosure 718 shouldact as a true plenum.

Referring now to FIG. 7D, airstream 720 likewise flows throughperforations 748 in the upper card-guide plate 704. The verticaldimension of the leaf-air outlet plenum 722 must be large enough toinsure that the flow through all perforations 748 is substantiallyequal; that is, enclosure 718 should act as a true plenum.

Referring now to FIG. 7F, in which all but one spine card 406 and oneleaf card 408 are hidden, the supply manifold 712 is supported by one ofthe plurality of columns 706. Supply hose 512 connects to supplymanifold 712 via a quick connect 750, and is routed to the liquid-cooledcold plate 510 using the vacant space at the corner of the spine card406. Routing of the return hose 514 is similar.

The leaf card 408, upon insertion, is guided into position by two cardsguides 752, one at the bottom, which is affixed to the lower card-guideplate 702, and the other at the top (not illustrated) which is affixedto the upper card-guide plate 704. In FIG. 7F, the lower card guides forhidden leaf cards are also hidden.

Referring now to FIGS. 8A-8C, precise location of each spine card in thez direction of coordinate system 410 (FIG. 4A) is assured, despitevariability in PCB thickness, by a plurality of standoffs 508 previouslyshown on FIGS. 5A-5E. Two instances of standoff 508, denoted 508A and508B, are shown on FIGS. 8A-8C; all reference numerals on the figuresare accordingly decorated with suffix A or B, but are referred to hereinwithout suffix. Each standoff 508 includes a shank 802 having a firstend face 804 in which is cut a threaded hole 806; a flange 808 having asecond end face 810; a first male-threaded section 812 having a thirdend face 814, where the outer diameter of section 812 is smaller thanthat of the flange 806; a second male-threaded portion 816 whose outerdiameter is smaller than that of the first male-threaded portion 812,and whose thread mates with that of the threaded hole 806; and a nut 818whose female thread mates with the male thread on the firstmale-threaded portion 812. In stacking the spine cards 408, a firstspine card 408A is captured between end face 810A and nut 818A, a secondspine card 408B is captured between end face 810B and nut 818B, andthreaded portion 816A is threaded into hole 806B until end face 814Aabuts end face 804B, as illustrated by the sequence shown in FIGS.8A-8C. In FIG. 8C, end faces 814A and 806B are abutted. Presuming thatall standoffs are substantially identical, the distance between endfaces 810A and 810B, which defines the pitch of the spine cards, isaccurately controlled to be equal to a length H that is the distancebetween end faces 804A and 814A of standoff 508A.

Referring now to FIGS. 9A-9E, assembling the stack of spine assemblies402 using standoffs 508 is accomplished upside down. First, a firstspine-card assembly 402A, complete with first-card standoffs 508A, isturned upside down, as illustrated in FIG. 9A, exposing, at the site ofeach first-card standoff 508A, the second male-threaded portion thereof,816A. Second, as illustrated in FIG. 9B, at each instance of afirst-card standoff 508A, a second-card standoff 508B is threaded ontightly, such that, referring to FIG. 8A, surface 804B of standoff 508Babuts surface 814A of standoff 508A. Third, as illustrated in FIG. 9C, asecond spine-card assembly 402B is placed upside down upon the firstthreaded portions 812B of standoffs 508B. Fourth, as illustrated in FIG.9D, nuts 818B are tightened securely onto threaded portions 812B,thereby capturing the second spine-card assembly 402B between nuts 818Band surfaces 810B. The second, third, and fourth steps recited above arethen repeated for each remaining spine-card assembly that needs to beadded to the stack. Finally, the stack is turned right side up. Theresult is shown in FIG. 9E, where the stack was built in the followingorder: A, B, C, D, E.

Referring now to FIGS. 10A and 10B, an alternative embodiment 1000 of ahigh-density, fail-in-place switch for computer and data networkscomprises a liquid-cooled spine-card array 1002, a first instance 1004Aof a liquid-cooled leaf-card array 1004 connected to the front of thespine-card array 1002, and a second instance 1004B of the leaf-cardarray 1004 connected to the rear of the spine-card array 1002. FIG. 10Ashows the switch 1000 assembled; FIG. 10B shows it exploded into thethree components 1002, 1004A, and 1004B.

Referring now to FIGS. 11A and 11B, the spine-card array 1002 includes aplurality of n_(S) spine-card assemblies 1102 (n_(S)=12 is shown) and aliquid-cooled cold plate 1104 that cools the spine-card assemblies. FIG.11A is a front view of array 1002; FIG. 11B is a rear view. The coldplate 1104, shown to best advantage in FIG. 11B, comprises an aluminumplate and embedded copper tubes, as is well known in the art. Otherdetails of spine-card cooling will be discussed presently.

Referring now to FIGS. 12A and 12B, the leaf-card array 1004 includes aplurality of n_(L)/2 leaf-card assemblies 1202 (n_(L)/2=12 is shown) anda liquid-cooled cold plate 1204 that cools the leaf-card assemblies.FIG. 12A is a front view of array 1004; FIG. 12B is a rear view. Thecold plate 1204, shown to best advantage in FIG. 12B, is constructed inthe same manner as cold plate 1104. Other details of leaf-card coolingwill be discussed presently.

Referring now to FIGS. 13A and 13B, a spine-card assembly 1102 includesa spine card 1302; a plurality of orthogonal headers 1304 arrayed on twoopposing edges of the spine card to convey electrical signals betweenthe spine card and leaf cards; a set of k_(S) spine chips 1306 (k_(S)=2is shown) (FIG. 13B); a power connector 1308 to convey power to thespine card; and for each spine chip, a cooling assembly that comprises aheat spreader 1310, a set of spine heat pipes 1312, and a cold block1314. A first end of each heat pipe 1312 is embedded in the heatspreader 1310, and a second end of each heat pipe is embedded in thecold block 1314. Consequently, heat is efficiently conveyed from theheat spreader 1310 to the cold block 1314, in a manner well known in theart of heat-pipe cooling. In this heat-pipe system, the heat spreader1310 serves as the evaporator and the cold block 1314 serves as thecondenser. Each cold block 1314 comprises an array of fasteners 1316,typically threaded fasteners, that removably attach the cold block 1314to the cold plate 1104 (see FIG. 11A). The cooling assemblies are hiddenin FIG. 13B to reveal the switch chips 1306.

Referring now to FIGS. 14A and 14B, a leaf-card assembly 1202 comprisesa leaf card 1402; a plurality of orthogonal receptacles 1404 arrayedalong a first edge of the leaf card 1402, one of which connects to eachspine card via mating headers 1304; a set of k_(L) leaf chips 1406(k_(L)=2 is shown) (FIG. 14B); a set of p_(L)/2 externally facing portconnectors 1408 (p_(L)/2=48 is shown) arrayed along a second edge of theleaf card 1402 that is opposite the first edge; and for each leaf chip,a cooling assembly that comprises a heat spreader 1410, a set of leafheat pipes 1412, and a cold block 1414. A first end of each heat pipe1412 is embedded in the heat spreader 1410, and a second end of eachheat pipe is embedded in the cold block 1414. Heat is efficientlyconveyed from the heat spreader 1410 to the cold block 1414, in a mannerwell known in the art of heat-pipe cooling. In this heat-pipe system,the heat spreader 1410 serves as the evaporator and the cold block 1414serves as the condenser. Each cold block 1414 comprises an array offasteners 1416, typically threaded fasteners, that removably attach thecold block 1414 to the cold plate 1204 (see FIG. 12A). The coolingassemblies are hidden in FIG. 14B to reveal the switch chips 1406. Theleaf-card assembly 1202 may additionally comprise cooling elements (notillustrated) for the port connectors 1408 in cases where the cablesterminating in these connectors produce heat, as in the case of activeoptical cables.

The fail-in-place strategy disclosed herein provides, for computer anddata networks, a switch having a large number of externally facingports. This is achieved by making an improved use of the perimeter ofthe spine cards for connection to leaf cards. Accessibility of spinecards is thereby sacrificed, precluding the easy repair thereof. Thistradeoff may be advantageous because the two-level, spine-leaf topologytypically used in switches, is inherently redundant, the fail-in-placestrategy disclosed herein causes only minor performance penalties, yetallows the number of externally facing ports to increase significantly,by as much as a factor of four compared to prior art, therebysignificantly increasing the number of computing and storage elementsthat may be interconnected without an undesirable cascading of switches.

While the description above contains much specificity, these should notbe construed as limitations on the scope, but rather as exemplificationsof preferred embodiments thereof. Many other variations are possible,such as, for example, the number of switch chips on the spine and leafcards may vary, the number of spine and leaf boards may vary, theconnections between spine and leaf boards may be optical as well aselectrical, and cooling of the chips may be accomplished in a variety ofways. The terminology used herein was chosen to best explain theprinciples of the embodiment, the practical application or technicalimprovement over technologies found in the marketplace, or to enableothers of ordinary skill in the art to understand the embodimentsdisclosed herein.

The invention claimed is:
 1. A network switch comprising: a plurality ofspine cards each comprising a liquid cooling plate in direct contactwith one or more spine chips, the plurality of spine cards are stackedvertically one on top of another and separated by a predetermined space;a plurality of leaf cards each comprising one or more leaf chips indirect contact with a heat sink, the plurality of leaf cards areperpendicular to and circumferentially arrayed around the stack of spinecards, the spine chips of each spine card are electrically connected tothe leaf chips of each leaf card, wherein the plurality of leaf cardsalong any one side of the stack of spine cards are arranged at an anglerelative to one another such that no two leaf cards along that side ofthe stack of spine cards are parallel; a heat exchanger comprisingsupply piping connecting it to the liquid cooling plate and returnpiping connecting it from the liquid cooling plate; lower card guideplate comprising perforations and guiding features corresponding witheach of the plurality of leaf cards; and upper card guide platecomprising perforations and guiding features corresponding with each ofthe plurality of leaf cards.
 2. The structure according to claim 1,wherein each leaf card is removably coupled to all of the spine cards.3. The structure according to claim 1, wherein an orthogonal connectorconnects all the spine cards to each of the leaf cards.
 4. The structureaccording to claim 1, wherein the leaf cards surround the stack of spinecards on all four sides, and all the leaf cards along each side of thestack of spine cards are separated from each other by a space at allfour corners of the stack of spine cards.
 5. The structure according toclaim 1, wherein the supply piping is routed through a space betweenadjacent leaf cards at a first corner of the stack of spine cards, andthe return piping is routed through a space between adjacent leaf cardsat a second corner of the stack of spine cards.
 6. The structureaccording to claim 1, wherein each of the one or more leaf chips arepositioned nearest to an innermost edge of each of the plurality of leafcards to minimize electrical path length in the leaf card.
 7. Thestructure according to claim 1, wherein each of the plurality of leafcards comprises a vapor chamber situated between the one or more leafchips and the heat sink to transfer the heat with low resistance from aninnermost edge of the leaf card to an outermost edge of the leaf card.8. The structure according to claim 1, wherein the heat sink comprisescooling fins.
 9. The structure according to claim 8, wherein a height ofthe cooling fins gradually increase from an innermost edge of the leafcard to an outermost edge of the leaf card to provide clearance foradjacent leaf cards.
 10. The structure according to claim 1, whereineach of the plurality of leaf cards comprises one orthogonal receptaclecorresponding to each spine card in the stack of spine cards.
 11. Thestructure according to claim 1, wherein each of the plurality of leafcards comprises multiple rows of RJ45 connectors arranged along an edgeopposite from the stack of spine cards.
 12. The structure according toclaim 1, wherein the supply piping is routed through a space betweenadjacent leaf cards at a first corner of the stack of spine cards, andthe return piping is routed through a space between adjacent leaf cardsat a second corner of the stack of spine cards.
 13. The structureaccording to claim 1, wherein each of the plurality of leaf cardscomprises a vapor chamber situated between the one or more leaf chipsand the heat sink to transfer the heat with low resistance from aninnermost edge of the leaf card to an outermost edge of the leaf card.14. A network switch comprising: a plurality of spine cards eachcomprising a liquid cooling plate in direct contact with one or morespine chips, the plurality of spine cards are stacked vertically one ontop of another and separated by a predetermined space; a plurality ofleaf cards each comprising one or more leaf chips in direct contact witha heat sink, the plurality of leaf cards are perpendicular to andcircumferentially arrayed around the stack of spine cards, the spinechips of each spine card are electrically connected to the leaf chips ofeach leaf card, wherein each of the plurality of spine cards comprisesone orthogonal receptacle corresponding to each of the plurality of leafcards, and, wherein each orthogonal receptacle along any one side ofeach of the plurality of leaf cards are arranged at an angle relative toone another such that no two orthogonal receptacles along that side ofeach of the plurality of leaf cards are parallel; a heat exchangercomprising supply piping connecting it to the liquid cooling plate andreturn piping connecting it from the liquid cooling plate; lower cardguide plate comprising perforations and guiding features correspondingwith each of the plurality of leaf cards; and upper card guide platecomprising perforations and guiding features corresponding with each ofthe plurality of leaf cards.
 15. The structure according to claim 14,wherein each leaf card is removably coupled to all of the spine cards.16. The structure according to claim 14, wherein an orthogonal connectorconnects all the spine cards to each of the leaf cards.
 17. Thestructure according to claim 14, wherein the leaf cards surround thestack of spine cards on all four sides, and all the leaf cards alongeach side of the stack of spine cards are separated from each other by aspace at all four corners of the stack of spine cards.
 18. The structureaccording to claim 14, wherein each of the one or more leaf chips arepositioned nearest to an innermost edge of each of the plurality of leafcards to minimize electrical path length in the leaf card.
 19. Thestructure according to claim 14, wherein the heat sink comprises coolingfins.
 20. The structure according to claim 19, wherein a height of thecooling fins gradually increase from an innermost edge of the leaf cardto an outermost edge of the leaf card to provide clearance for adjacentleaf cards.
 21. The structure according to claim 14, wherein each of theplurality of leaf cards comprises one orthogonal receptaclecorresponding to each spine card in the stack of spine cards.
 22. Thestructure according to claim 14, wherein each of the plurality of leafcards comprises multiple rows of RJ45 connectors arranged along an edgeopposite from the stack of spine cards.